Biases in the Measurement of Hidden Populations

A Meta-Analysis of PRIF Studies

Georgiy Syunyaev

WZB Berlin

Macartan Humphreys

WZB Berlin

May 23, 2023

Outline

  • Punchlines
  • Setup
  • Analysis Harmonization & Replication: hiddenmeta [R] package
  • Priors
  • Meta-analysis model: Homogeneous biases
  • Meta-analysis model: Heterogeneous biases
  • Cost-benefit analysis
  • Dashboard
  • Wrap up

Punchlines

  • From priors survey, many expected PPS / HT to do well and other approaches to underestimate
  • In fact PPS / HT seems to dominate both in terms of MSE and cost
  • Biases appear not to be homogeneous across studies, counter to our model
  • Model that allows heterogeneity suggests that heterogeneity in relative bias of methods is large
  • Models provide adjusted estimates for each team: these are mostly similar across models, though assumption dependent
  • We still have relatively weak overlap in methods across studies: this means still some sensitivity to priors especially for heterogeneity model

Set up

PRIF meta-analysis

  • Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

PRIF meta-analysis

  • Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

  • Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs

PRIF studies

  • Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs
Implementing organization Location Hard-to-reach group (numerator) Population (denominator)
Freedom Fund (FF) Brazil (Recife) Women 18-21 who were sex workers before age 18 Women in age 18-21 who are sex workers
Stanford University (Stanford) Brazil Forced labor in agriculture sector Employed in agriculture sector
New York University (NYU) Costa Rica Forced labor in fishing industry Employed in fishing industry
New York University (NYU) Tanzania (Dar es Salaam, Iringa, and Zanzibar) Domestic servitude Domestic workers
NORC Morocco Domestic servitude Domestic workers
Johns Hopkins University (JHU) Pakistan (Sindh) Forced labor in brick kilns industry Employed in brick kilns industry
University of Massachusetts, Lowell (UMass) Tunisia (Tunis) Domestic servitude All domestic workers
RTI USA Forced labor in construction industry All construction workers

PRIF meta-analysis

  • Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

  • Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs

  • Approach: Use MIDA to provide both meta-analysis of PRIF studies and simulations for study design diagnosis

MIDA

  • Approach: Use MIDA to provide both meta-analysis of PRIF studies and simulations for study design diagnosis
flowchart TD
  A{{"Model (Features of unobserved population)"}} === B{{"Inquiries (Quantities of interest)"}}
  B === C{{"Data strategy (Sampling strategies used)"}}
  C === D{{"Answer strategy (Estimation methods used)"}}

PRIF meta-analysis

  • Question: What are the relative biases of different methods for estimation of prevalence and size of hard-to-reach populations?

  • Setting: 8 PRIF studies of prevalence of various types of human-trafficking around the world using somewhat harmonized designs

  • Approach: Use MIDA to provide both meta-analysis of PRIF studies and simulations for study design diagnosis

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta

Analysis Harmonization & Replication

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta

Following MIDA:

  1. Model (Population)
  2. Inquiries
  3. Data strategies
  4. Answer strategies

Model (Population)

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
  1. Population network simulated with get_study_population()

Inquiries

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
  1. Population network simulated with get_study_population()
  2. Estimands calculated with get_study_estimands()
Study-level estimands
Inquiry Estimand
known_size 105.000
known_prev 0.105
hidden_size 202.000
hidden_prev 0.202

Data strategies

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
  1. Population network simulated with get_study_population()
  2. Estimands calculated with get_study_estimands()
  3. Sampling simulated with sample_rds() / sample_pps() / sample_tls()

Data strategies

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
  1. Population network simulated with get_study_population()
  2. Estimands calculated with get_study_estimands()
  3. Sampling simulated with sample_rds() / sample_pps() / sample_tls()

Data strategies

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
  1. Population network simulated with get_study_population()
  2. Estimands calculated with get_study_estimands()
  3. Sampling simulated with sample_rds() / sample_pps() / sample_tls()

Answer strategies

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
  1. Population network simulated with get_study_population()
  2. Estimands calculated with get_study_estimands()
  3. Sampling simulated with sample_rds() / sample_pps() / sample_tls()
  4. Estimates from samples with get_study_est_*()

Answer strategies: Prevalence

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
Prevalence estimation methods
Sampling Estimation Method references [R] Package
PPS HT: Horvitz-Thompson estimator of prevalence with re-scaled bootstrap standard errors Rust and Rao (1996) Self-coded + surveybootstrap (Feehan and Salganik 2023)
TLS HT: Horvitz-Thompson estimator of prevalence with re-scaled bootstrap standard errors (Rust and Rao 1996) Rust and Rao (1996) Self-coded + surveybootstrap (Feehan and Salganik 2023)
RDS SS: Sequential sampling estimator of prevalence Gile (2011) RDS (Mark S. Handcock et al. 2023)
RDS+/LTS LINK: Link-tracing prevalence estimator based on snowball sample Vincent and Thompson (2022) Self-coded based on Vincent and Thompson (2022)

Answer strategies: Population size

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
Population size estimation methods
Sampling Estimation Method references [R] Package
PPS NSUM: Network scale-up estimator of population size with re-scaled bootstrap standard errors Killworth et al. (1998); Rust and Rao (1996) networkreporting (Feehan and Salganik 2016) + surveybootstrap (Feehan and Salganik 2023)
TLS NSUM: Network scale-up estimator of population size with re-scaled bootstrap standard errors (Killworth et al. 1998; Rust and Rao 1996) Killworth et al. (1998); Rust and Rao (1996) networkreporting (Feehan and Salganik 2016) + surveybootstrap (Feehan and Salganik 2023)
TLS RECAP *: Mark-recapture estimator of population size with parametric standard errors Hickman et al. (2006) Rcapture (Baillargeon and Rivest 2007)
RDS SSPSE: Bayesian sequential sampling model of population size Mark S. Handcock, Gile, and Mar (2014) sspse (Mark S. Handcock et al. 2022)
RDS CHORDS: Epidemiological model for population size estimation Berchenko, Rosenblatt, and Frost (2017) chords (Berchenko, Rosenblatt, and Frost 2017)
RDS MULTIPLIER *: Service-multiplier population size estimator with bootstrap re-sampling standard errors Hickman et al. (2006); Salganik (2006) Self-coded + surveybootstrap (Feehan and Salganik 2023)
RDS+/LTS LINK: Link-tracing population size estimator based on snowball sample Vincent and Thompson (2022) Self-coded based on Vincent and Thompson (2022)
RDS+/LTS MSE: Link-tracing population size estimator based on multiple link-tracing samples Vincent and Thompson (2017) Self-coded based on Vincent and Thompson (2017)

Answer strategies: Comparison with study estimates

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta

  • Running estimation implemented in the hiddenmeta package is straightforward…

    • …once data is in shape. Currently data is in shape for this for two teams.
Example code for estimation
hiddenmeta::get_study_est_ss(
      data = jhu_pakistan,
      sampling_frame =  "known1",
      hidden_var = "hidden",
      total_var = "total",
      n_coupons = 3,
      prefix = "rds",
      label = "rds_ss")


hiddenmeta::get_study_est_ht(
      data = umass_tunisia,
      hidden_var = "hidden",
      survey_design = ~ tls_cluster,
      weight_var = "tls_weight",
      prefix = "tls",
      label = "tls_ht")
  • The results do not always align with what teams produced…
    • …possibly due to data re-coding or model fine-tuning
Comparison of estimates from hiddenmeta to study estimates
Estimates
Std. Errors
Study Inquiry Sample Estimator Team Replication Team Replication
jhu_pakistan hidden_prev2 rds ss 0.255 0.457 0.023 0.04
jhu_pakistan hidden_size2 pps ht 27400 3208
jhu_pakistan hidden_prev2 pps ht 0.125 0.196 0.019 0.023
umass_tunisia hidden_size2 tls ht 1383 208
umass_tunisia hidden_prev2 tls ht 0.238 0.22 0.013 0.033
umass_tunisia hidden_size2 tls recap 3148 870
umass_tunisia hidden_size2 tls mse 3360 1048

Meta-analysis models

Meta-level

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
data {
  int<lower=0> N;
  int<lower=0> K;
  int<lower=0> n_ests;
  real<lower=0> alpha_prior_mean[N];
  real<lower=0> alpha_prior_sd[N];
  real<lower=0> bias_prior_mean[K];
  real<lower=0> bias_prior_sd[K];
  int<lower=0> site[n_ests];
  int<lower=0> design[n_ests];
  vector<lower=0>[n_ests] ests;
  vector<lower=0>[n_ests] ses;
}
parameters { }
model { }

Meta-level: Stan model

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
data {
  int<lower=0> N;
  int<lower=0> K;
  int<lower=0> n_ests;
  real<lower=0> alpha_prior_mean[N];
  real<lower=0> alpha_prior_sd[N];
  real<lower=0> bias_prior_mean[K];
  real<lower=0> bias_prior_sd[K];
  int<lower=0> site[n_ests];
  int<lower=0> design[n_ests];
  vector<lower=0>[n_ests] ests;
  vector<lower=0>[n_ests] ses;
}
parameters {
  vector<lower=0.01>[K] beta;
  vector<lower=0,upper=1>[N] alpha;
}
model { }

Meta-level: Stan model

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
data {
  int<lower=0> N;
  int<lower=0> K;
  int<lower=0> n_ests;
  real<lower=0> alpha_prior_mean[N];
  real<lower=0> alpha_prior_sd[N];
  real<lower=0> bias_prior_mean[K];
  real<lower=0> bias_prior_sd[K];
  int<lower=0> site[n_ests];
  int<lower=0> design[n_ests];
  vector<lower=0>[n_ests] ests;
  vector<lower=0>[n_ests] ses;
}
parameters {
  vector<lower=0.01>[K] beta;
  vector<lower=0,upper=1>[N] alpha;
}
model {
  target += normal_lpdf(ests | beta[design].*alpha[site], ses);
  alpha ~ normal(alpha_prior_mean, alpha_prior_sd);
  beta ~ lognormal(log(beta_prior_mean), beta_prior_sd);
}
  • likelihood of a given estimate \(\hat{\alpha}_{ij}\) for estimand \(\alpha_i\) using method \(j\) in location \(i\) is \[\phi(\hat{\alpha}_{ij}|\beta_j\alpha_{i}, \hat{\sigma}_{ij})\]

Note: we assume that the standard errors correctly estimate the standard deviation of the sampling distribution of the estimate (even if the estimate is unbiased)

Meta-level: Stan model

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
data {
  int<lower=0> N;
  int<lower=0> K;
  int<lower=0> n_ests;
  real<lower=0> alpha_prior_mean[N];
  real<lower=0> alpha_prior_sd[N];
  real<lower=0> bias_prior_mean[K];
  real<lower=0> bias_prior_sd[K];
  int<lower=0> site[n_ests];
  int<lower=0> design[n_ests];
  vector<lower=0>[n_ests] ests;
  vector<lower=0>[n_ests] ses;
}
parameters {
  vector<lower=0.01>[K] beta;
  vector<lower=0,upper=1>[N] alpha;
}
model {
  target += normal_lpdf(ests | beta[design].*alpha[site], ses);
  alpha ~ normal(alpha_prior_mean, alpha_prior_sd);
  beta ~ lognormal(log(beta_prior_mean), beta_prior_sd);
}
  • Prior on \(\alpha\) (prevalence/population size) is \[\alpha_i \sim f(\mu_{\alpha, i}, {\sigma}_{\alpha, i}),\] where the \(\mu_{\alpha, i}\) is gathered from teams, \(\sigma_{\alpha, i}\) is large, and \(f\) denotes the normal distribution.
  • Note that we assume no relation between different elements of \(\alpha\)—that is we do not make use of a prior distribution on prevalence “in general.”

Meta-level: Stan model

  • Implementation: Provide a well documented package that implements sampling and estimation strategies (both at study and meta levels): gsyunyaev.com/hiddenmeta
data {
  int<lower=0> N;
  int<lower=0> K;
  int<lower=0> n_ests;
  real<lower=0> alpha_prior_mean[N];
  real<lower=0> alpha_prior_sd[N];
  real<lower=0> bias_prior_mean[K];
  real<lower=0> bias_prior_sd[K];
  int<lower=0> site[n_ests];
  int<lower=0> design[n_ests];
  vector<lower=0>[n_ests] ests;
  vector<lower=0>[n_ests] ses;
}
parameters {
  vector<lower=0.01>[K] beta;
  vector<lower=0,upper=1>[N] alpha;
}
model {
  target += normal_lpdf(ests | beta[design].*alpha[site], ses);
  alpha ~ normal(alpha_prior_mean, alpha_prior_sd);
  beta ~ lognormal(log(beta_prior_mean), beta_prior_sd);
}
  • Prior on \(\beta_j\) (bias) is \[\beta_j \sim g(\mu_{\beta_j}, {\sigma}_{\beta_j}),\] where the \(\mu_{\beta_j}\) is gathered from teams, \(\sigma_{\beta_j}\) is generally large, and \(g\) denotes the log-normal distribution.
  • A tighter variance can be placed on any “benchmark” approach
  • We assume no relation between different elements of \(\beta\) – that is we do not make use of a prior distribution about biases of different methods “in general.”

Priors

Disagreements about bias

Distribution of expected bias across respondents

  • Most respondents expected that most methods would underestimate prevalence
  • Some expected rds would especially underestimate prevalence
  • One expected all methods would be unbiased
  • PPS-HT has the tightest distribution centered close to 1

Uncertainty about bias

Average bias expectation with average lower and upper bounds of credibility intervals

  • There is considerable ex ante uncertainty regarding the performance of methods
  • In almost all cases “average bounds” include 1
  • Only for TLS-NSUM was the “average” credibility interval entirely above 1

Main meta-analysis results

Data

  • Sample of estimates depends on estimands: prevalence (threshold 1 / threshold 2 / both), size (hidden / known)
    • e.g. for prevalence (threshold 1)
Study LTS-HCG LTS-NE4NS LTS-NE4NS2 LTS-VH PPS-HT RDS-SS TLS-HT TLS-RMARK
Brazil (FF) x
Pakistan (JHU) x x
Morocco (NORC) x x
Costa Rica (NYU) x x x x x
Tanzania (NYU) x x x x x
Tunisia (UMass) x x

Main meta-analysis results: Threshold 1 (Biases)

prif_estimates |> 
  filter(inquiry == "hidden_prev1") |> 
  get_meta_estimates()

Main meta-analysis results: Threshold 1 (Prevalence)

Main meta-analysis results: Threshold 2 (Biases)

prif_estimates |> 
  filter(inquiry == "hidden_prev2") |> 
  get_meta_estimates()

Main meta-analysis results: Threshold 2 (Prevalence)

Estimated bias vs. priors

Estimated bias vs. bias priors

  • Estimated biases exhibit more variance than priors on biases

Alternative model:
Allow arbitrary heterogeneity

Approach

Instead of assuming that the relative bias of a method \(\beta_{j}\) is constant, we assume that at each site it is a draw from a distribution: \[\beta_{j} \sim f(\beta^*_{j}, \sigma_{j})\] We get to estimate both \(\beta^*_{j}\) – the expected relative bias, and \(\sigma_{j}\): the heterogeneity in relative bias.

Bias comparisons across models

Estimated bias comparisons

  • These correlate. Basic message though is that the heterogeneous model is consistent with strong heterogeneity: We entertain the possibility that every method could be an overestimate or an underestimate.

  • This model has many parameters for not so many data points \(\Rightarrow\) prior sensitivity

Prevalence estimates from two models

Prevalence estimates from two models

  • Biggest difference is in Brazil which sees less adjustment in heterogeneous model

Cost benefit analysis

  • Plot estimates of root mean squared error (RMSE) against approximate costs of each sampling strategy provided by teams to identify strategies that are undominated wrt costs and error.
  • Interested in assessing the cost-error frontier
  • To estimate RMSE we use the expression for mean square error:

\[ \mathrm{MSE}(\hat\beta) = \mathrm{Var}(\hat\beta) + \mathrm{Bias}(\hat\beta,\beta)^2 \]

  • To assess Bias we use the difference between the posterior prevalence estimate and the raw estimate in question.
Study Type of sample Sample size Costs (in 1000 USD)
Brazil (FF) PPS 1000 50
Brazil (FF) RDS 600 100
Pakistan (JHU) PPS 800 45
Pakistan (JHU) RDS 500 65
Morocco (NORC) LTS/RDS+ 989 350
Morocco (NORC) TLS/TLS-RECAP 1067 450
Costa Rica (NYU) PPS 1017 79
Costa Rica (NYU) LTS/RDS+ 1009 212
Tanzania (NYU) LTS/RDS+ 788 15
Tanzania (NYU) PPS 1052 55
Tunisia (UMass) TLS-RECAP 1016 80
Tunisia (UMass) TLS 1029 100

Cost-benefit analysis by study

  • Note that the frontier is covered largely by PPS. LTS/RDS+ (Tanzania) is also on the frontier here with low cost, but large error (from our estimates).

Cost-benefit analysis by strategy

Dashboard

Interactive meta-analysis

  • The meta-analysis results are preliminary:
    • Estimates from some teams were not included in the meta-analysis dataset
    • The costs of each method might need to be adjusted
    • The priors on study level inquiries as well as relative biases might need to be adjusted
  • We built an interactive Shiny dashboard for everyone to be able to explore and adjust data and priors
  • We encourage everyone to access the dashboard and play with it here: shiny2.wzb.eu/syunyaev/hiddenmetashiny/

Interactive meta-analysis

Wrap up

Worries and todos


Todos

  • Replication work not completed: please bring your data!
  • Still relatively low overlap (\(n\) per method); we want to improve this (bring your data and lets implement the NSUMs)
  • Implement structured heterogeneity analysis: for instance, based on types of networks

Worries and issues

  • We are seeing some variation in results as a function of thresholds
  • Seems clear we need to deviate from PAP
  • We are assuming independence of samples in each study but this is not right

Onwards


  • We are setting up the dashboard so that you can add more data and the meta-analysis updates automatically.

  • The dream is as you do more studies you use multiple methods and you feed into this “rolling meta-analysis”

    • Who will curate?
    • What documentation is needed to join the club?

Conclusion and reflections


High level conclusion

  • We know different people here love RDS, Link tracing, and TLS
  • There are good prior reasons for each of these
  • But meta-analysis suggests HT (more than) competitive
    • Others seem to underestimate in general
    • Others appear costly

Model performance

  • Bias appear to be demonstrably not homogeneous across studies, counter to our pre-registered model
  • Sensitivity to priors, thresholds

Reflections


  • Even if estimates similar, if these are getting at different parts of the population then these are not substitutable: approach to get prevalence of subgroups within target population
  • What finer data to use to assess optimal “mixing”?
  • What else to take into account for decision between methods (e.g. Pakistan team described RDS as safer–this not captured in our analysis. )

Questions

References

Baillargeon, Sophie, and Louis-Paul Rivest. 2007. “Rcapture: Loglinear Models for Capture-Recapture in r.” Journal of Statistical Software 19: 1–31.
Berchenko, Yakir, Jonathan D. Rosenblatt, and Simon D. W. Frost. 2017. “Modeling and Analyzing Respondent-Driven Sampling as a Counting Process.” Biometrics 73 (4): 1189–98. https://doi.org/10.1111/biom.12678.
Feehan, DM, and MJ Salganik. 2016. “Networkreporting: Tools for Using Network Reporting Estimators [r Package Version 0.1.1].”
———. 2023. “Surveybootstrap: Tools for the Bootstrap with Survey Data [r Package Version 0.0.3].”
Gile, Krista J. 2011. “Improved Inference for Respondent-Driven Sampling Data with Application to HIV Prevalence Estimation.” Journal of the American Statistical Association 106 (493): 135–46.
Handcock, Mark S., Krista J. Gile, and Corinne M. Mar. 2014. “Estimating Hidden Population Size Using Respondent-Driven Sampling Data.” Electronic Journal of Statistics 8 (1): 1491–521. https://doi.org/10.1214/14-EJS923.
Handcock, Mark S, Krista J Gile, Ian E Fellows, and W Whipple Neely. 2023. “RDS: Respondent-Driven Sampling Package.”
Handcock, Mark S, Krista J Gile, Brian Kim, and Katherine R McLaughlin. 2022. “Sspse: Estimating Hidden Population Size Using Respondent Driven Sampling.”
Hickman, Matthew, Vivian Hope, Lucy Platt, Vanessa Higgins, Mark Bellis, Tim Rhodes, Colin Taylor, and Kate Tilling. 2006. “Estimating Prevalence of Injecting Drug Use: A Comparison of Multiplier and Capture-Recapture Methods in Cities in England and Russia.” Drug and Alcohol Review 25 (2): 131–40.
Killworth, Peter D, Eugene C Johnsen, Christopher McCarty, Gene Ann Shelley, and H Russell Bernard. 1998. “A Social Network Approach to Estimating Seroprevalence in the United States.” Social Networks 20 (1): 23–50.
Rust, Keith F, and JNK Rao. 1996. “Variance Estimation for Complex Surveys Using Replication Techniques.” Statistical Methods in Medical Research 5 (3): 283–310.
Salganik, Matthew J. 2006. “Variance Estimation, Design Effects, and Sample Size Calculations for Respondent-Driven Sampling.” Journal of Urban Health 83 (Suppl 1): 98.
Vincent, Kyle, and Steve Thompson. 2017. “Estimating Population Size with Link-Tracing Sampling.” Journal of the American Statistical Association 112 (519): 1286–95.
———. 2022. “Estimating the Size and Distribution of Networked Populations with Snowball Sampling.” Journal of Survey Statistics and Methodology 10 (2): 397–418.